Markov Decision Processes with General Discount Functions

In Markov Decision Processes, the discount function determines how much the reward for each point in time adds to the value of the process, and thus deeply a ects the optimal policy. Two cases of discount functions are well known and analyzed. The rst is no discounting at all, which correspond to the totaland average-reward criteria. The second case is a constant discount rate, which leads to a decreasing exponential discount function. However, other discount functions appear in many models, including those of human decisionmaking and learning, making it interesting and possibly useful to investigate other functions. We review results for a weighted sum of several discount functions with di erent cost functions, showing that nite models with this criterion have optimal policies which are stationary from a xed time N, aptly called Nstationary. We review a proof for their existence and an algorithm for their computation, as well as remark on the structure of these policies as the discount factors vary. We then discuss two attempts to generalize the results for weighted exponential discount functions. The rst is a hypothesis for a sum of di erent general discount function with certain exponential bounds, in the spirit of the results for the exponential case. We show via counterexample that despite the intuitive appeal of the hypothesis, it is in fact not true, and make some remarks on why this is so. Our second attempt at generalization is to represent a general discount function as an in nite sum of decreasing exponential functions with constant coe cients. We give convergence conditions on the sum under which the previously established results can be extended to enable us to nd an optimal policy for it. We discuss two examples that clarify our results, and connect them to areas which require non-exponential discount functions. The work is concluded by an example of a model with a monotonic discount function that has no optimal N-stationary policy.